Identifying Unreliable and Adversarial Workers in Crowdsourced Labeling Tasks

In this paper, we study the problem of aggregating noisy responses from crowd workers to infer the unknown true labels of binary tasks. Unlike most prior work which has examined this problem under the probabilistic worker paradigm, we consider a much broader class of adversarial workers with no specific assumptions on their labeling strategy. Our key contribution is the design of a computationally efficient reputation algorithm to identify and filter out such adversarial workers in crowdsourcing systems, given only the labels provided by the workers. Our algorithm uses the concept of optimal semi-matchings in conjunction with worker penalties based on label disagreements, to detect outlier worker labeling patterns. We prove that our algorithm can successfully identify low reliability workers, workers adopting deterministic strategies; and is robust to manipulation by worstcase sophisticated adversaries who can adopt arbitrary labeling strategies to degrade the accuracy of the inferred task labels. Finally, we show that our reputation algorithm can significantly improve the accuracy of existing label aggregation algorithms in real-world crowdsourcing datasets.